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Hie  objective  of  this  study  was  to  investigate,  describe  and  analyze  the  key 
factors  surrounding  the  establishment  of  the  Air  Force  Test  and  Evaluation 
Center  (AFTEC) , the  Operational  Test  and  Evaluation  Agency  (OTEA)  and  the 
Operational  Test  and  Evaluation  Force  (OPIEVFOR) . 


Data  sources  used  were  of  two  types.  First,  a literature  search  was  conducted 
to  obtain  as  much  background  information  on  the  subject  as  possible.  Secondly, 
interviews  were  conducted  with  key  personnel  of  the  independent  test  agencies 
and  various  DOD  Program  Management  Office  personnel,  some  of  which  are  attending 
the  Defense  Systems  Management  School. 

The  information  obtained  from  all  sources  is  presented  and  conclusions  are 
drawn  that  current  QT&E  is  extremely  costly  and  that  there  may  be  ways  to 
reduce  these  costs.  Three  significant  reccmrendations  are  presented. 
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EXECUTIVE  SUMMARY 

In  the  1970-71  time  period.  President  Nixon's  Blue  Ribbon 
Defense  Panel  (BRDP)  and  the  U.S.  Congress  became  deeply  involved 
in  DOD  weapon  system  test  and  evaluation  (T&E)  policy  and  inde- 
pendence. The  independence  they  addressed  was  separation  of  the 
system  tester  from  the  developer  and  user,  both  of  which  they 
considered  system  advocates.  Further,  neither  Washington  group 
felt  that  sufficient  testing  was  being  accomplished  prior  to  key 
acquisition  decision  points.  As  a result  of  this  high  level 
criticism,  OSD  directed  the  three  Services  to  establish  independ- 
ent test  agencies  (ITA's)  and  to  conduct  more  and  improved  T&E. 

The  purpose  of  this  study  was  to  learn  more  about  the  causes 
for  and  results  of  that  OSD  direction.  The  review  included  a 
survey  of  official  government  records  such  as  the  BRDP  Report  and 
sources  containing  Congressional  testimonies.  Significant  excerpts 
from  these  documents  can  be  found  in  Section  II. 

The  Services  established  their  ITA's  during  the  period  1972- 
1974.  Section  III  reports  on  the  agencies  themselves,  including 
such  factors  as  operational  dates,  organization  structure  and 
manning.  Comparisons  and  contracts  are  made  between  the  Army, 

Navy  and  Air  Force  ITA's.  In  addition,  the  current  mission, 
number  of  major /non-major  test  programs  and  OT&E  management  poli- 
cies of  each  agency  are  reviewed. 


ii 


One  initial  and  major  objective  of  this  project  was  to  in- 
vestigate and  compare  costs  of  OT&E  before  and  after  the  intro- 
duction of  ITA's.  However,  this  objective  was  modified  in  ''mid- 
stream” when  it  became  apparent  that  actual  T&E  costs  are  just 
not  available.  Section  IV  addresses  this  cost  accountability 
problem  and  provides  a look  at  current  ITA  budgeting  and  funding. 
In  addition,  it  introduces  some  of  the  non-dollar  costs  of  oper- 
ating under  the  ITA  concept. 

Sections  V and  VI  summarize  the  study  and  conclude  that 
current  OT&E  costs  are  very  high.  There  is  evidence  of  "over- 
testing"  and  loss  of  program  control  by  the  DOD  Program  Manager. 

Three  recommendations  pertinent  to  reducing  costs  of  OT&E 
are  presented  in  Section  VII. 
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In  this  Bicentennial  year  when  all  Americans  are  thinking 
of  and  frequently  hearing  the  word  "independence",  it  seemed  be- 
fitting to  address  a recent  major  change  to  the  U.S.  military's 
systems  acquisition  process  - Independent  Test  Agencies  (ITA's). 

Over  the  past  half  decade,  much  has  been  written,  stated 
and  directed  on  the  need  for  and  benefits  of  independent  testing 
in  the  DOD  systems  acquisition  process.  Since  the  July  1970 
Blue  Ribbon  Defense  Panel  Report,  Test  and  Evaluation  (T&E)  poli- 
cies of  the  military  services  have  been  significantly  altered. 
Separate  independent  test  agencies  (ITA's)  have  been  established 

by  the  Army,  Navy  and  Air  Force.  These  agencies,  although  organ- 
* 

ized  somewhat  differently  in  their  individual  Service's  hierarchy, 
have  one  fundamental  element  in  common;  each  maintains  its  testing 
independence  from  both  the  developer  and  the  user,  reporting 
directly  to  Service  headquarters.  The  three  ITA's  vary  consider- 
ably in  size  from  the  Air  Force's  organization  of  approximately 
230  people  to  ha  Navy's  OPTEVFOR  which  numbers  over  1,400  offi- 
cers, enlisted  and  civilian  personnel. 

It  is  extremely  dirficult  to  take  issue  with  the  theory  that 
independent  testing  and  reporting  provides  increased  objectivity 
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to  the  systems  acquisition  process.  Likewise,  one  would  be 
foolish  to  disagree  with  the  fact  that  objective  testing  and 
evaluation  are  essential  to  proper  decision  making  throughout 
all  phases  of  systems  acquisition.  The  point  in  the  process 
where  test  results  appear  to  be  most  critical  (in  terms  of  dol- 
lar decisions  and  high  level  review)  is  at  Defense  System  Acqui- 
sition Review  Council  (DSARC)  III.  It  is  from  this  review  that 
the  Secretary  of  Defense  determines  whether  to  approve  or  deny 
production  go-ahead  for  the  system.  This  decision,  which  nor- 
mally follows  the  completion  of  full- scale  development  opera- 
tional testing,  draws  heavily  upon  the  demonstrated  system  per- 
formance data  as  reflected  in  the  reports  by  the  ITA's.  These 
agencies  exist  today  primarily  because  of  the  significance  of 
the  DSARC  III  decision  in  terms  of  the  nation’s  defense  posture 
and  budget. 

Because  two  of  the  three  service  ITA's  are  relatively  young 
organizations  and  the  third  has  just  recently  incorporated  inde- 
pendent operational  testing,  their  contributions  to  date  are 
difficult  to  measure.  No  "track  records"  are  available  since 
no  weapon  systems  have  yet  been  deployed  for  which  the  ITA's 
have  had  full  test  management  responsibility.  Future  analysis 
of  their  worth  should  address  items  such  as:  Are  the  systems 

being  fielded  by  the  services  more  dependable,  better  performing. 


and/or  less  costly  to  operate  and  maintain?  If  they  are,  how 
much  of  the  increased  success  can  be  attributed  directly  or 
indirectly  to  the  activities  of  the  ITA's?  And  finally,  what 
has  been  the  cost  of  the  increased  system  effectiveness  due  to 
ITA  involvement? 

OSD,  when  directinq  Service  ITA's,  was  convinced  that  in- 
dependent testing  was  essential  to  the  decision-making  processes 
of  weapon  systems  acquisition. 

This  paper  reviews  the  issues  and  activities  leading  to 
that  direction,  discusses  how  each  service  implemented  that  dir- 
ection, and  addresses  some  of  the  resulting  costs.  The  costs 
examined  are  not  limited  to  those  items  that  can  bte  identified 
on  a budget  or  other  financial  document.  Just  as  important  are, 
those  factors  more  difficult  for  a program  manager  to  quantify 
such  as  system  responsibility,  program  control  and  schedule. 


&tzrsmsil « — 


mf 


r msk:vi> . V Vi^^ivaxwas^3bc«25c« 


SECTION  II 
BACKGROUND 

Previous  Operational  Testing 

Until  quite  recently,  operational  testing  of  new  weapon  and 
other  military  systems  was  conducted  and  reported  on  by  the  im- 
plementing (developer)  and  using  commands.  Now  and  into  the 
foreseeable  future,  separate  agencies  within  each  service  are  and 
will  be  responsible  for  managing  or  monitoring  all  operational 
testing. 


DT&E  versus  OT&E 


Before  proceeding  with  the  causes  for  this  drastic  change 
in  T&E  policy  and  procedure,  a brief  differentiation  between 
operational  testing  and  development  testing  is  in  order.  Devel- 
opment test  and  evaluation  (DT&E)  is  concerned  primarily  with 
the  engineering  function  of  the  system  design. 

DT&E  is  conducted  to  verify  accomplishment  of 
the  development  objectives;  these  include  refining 
engineering  design,  verifying  that  design  risks  have 
been  minimized,  demonstrating  that  the  system  will 
meet  specifications,  and  estimating  the  military 
utility.  DT&E  is  essentially  a detailed  engineering 
analysis  of  a system's  performance,  beginning  with 
individual  subsystems  and  progressing  through  a com- 
plete system,  where  system  design  is  tested  and 
evaluated  against  engineering  and  operational  criteria 
by  the  implementing  command...  (1) 
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On  the  other  hand,  operational  test  and  evaluation  (OT&E)  focuses 
on  the  intended  use  of  the  system  in  its  operational  environment . 

OT&E  is  conducted  to  estimate  the  system's 
military  utility,  operational  effectiveness,  operational 
suitability,  and  to  identify  deficiencies.  In  addition, 

OT&E  provides  information  on  organization,  personnel 
requix ements , doctrine,  and  tactics.  OT&E,  normally 
composed  of  initial  OT&E  (IOT&E)  and  follow-on  OT&E 
(FOT&E) , is  essentially  an  operational  analysis  of 
a system's  performance  where  the  complete  system  is 
tested  and  evaluated  against  operational  criteria  by 
personnel  with  the  same  qualifications  as  those  who 
will  use,  maintain  and  support  the  system  when  deployed.  (1) 


What  Prompted  ITA's? 

During  the  period  1950-1970,  defense  system  planners  and 
developers  received  "little  flak"  from  higher  levels  of  the 
government  bureaucracy  on  the  methods  being  used  to  develop  and 
test  new  systems..  Until  then,  most  controversy  surrounded  the 
actual  need  for  the  system  in  the  Defense  inventory  and  the  cost 
of  putting  it  there.  Concern  over  whether  a system  was  actually 
ready  for  production  or  performed  to  certain  specifications  was 
minimal  outside  DOD.  This  lack  of  concern  is  attributed  to  a 
general  satisfaction  within  congressional  and  executive  arenas 
that  government  Research,  Development,  Test  and  Evaluation  (RDT&E) 
activities  had  been  successful  in  maintaining  U.S.  military 
superiority,  in  winning  the  ICBM  and  space  race  and  putting  man 
on  the  moon.  In  addition,  there  were  factors  such  as  the  Vietnam 
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Conflict  which  had  caused  DOD  to  defer  many  major  weapon  system 
acquisition  programs.  As  the  1970's  approached,  economic  condi- 
tions and  domestic  issues  such  as  social  programs  and  environ- 
mental improvement  caused  Congress  to  intensify  its  scrutiny  of 
DOD  expenditures,  especially  in  the  area  of  major  weapon  systems 
acquisition.  As  a result  of  this  intensification  on  the  part  of 
Congress  and  the  President,  T&E  policies  of  the  military  services 
would  change  dramatically. 

The  July  1970  Blue  Ribbon  Defense  Panel  Report  (also  known 
as  the  Fitzhugh  Report)  was  the  driving  force  behind  this  change. 
The  panel,  formed  by  President  Nixon  to  examine  the  Department  of 
Defense,  made  a searching  review  of  DOD  systems  acquisition 
policies,  including  T&E.  The  following  critical  judgment  con- 
cerned DOD  Operational  Test  and  Evaluation  (OT&E)  activities: 

There  has  been  an  increasing  desire,  particularly 
at  OSD  level,  to  use  data  from  OT&E  to  assist  in  the 
decision-making  process.  Unquestionably,  it  would  be 
extremely  useful  to  replace  or  support  critical  assump- 
tions and  educated  guesses  with  quantitative  data  ob- 
tained from  realistic  and  relevant  operational  testing. 

Unfortunately,  it  has  been  almost  impossible  to 
obtain  results  which  are  directly  applicable  to  decision 
or  useful  for  analysis.  Often,  test  data  do  not  exist. 

When  they  do,  they  frequently  are  derived  from  tests 
which  were  poorly  designed  or  conducted  under  insuffi- 
ciently controlled  conditions  to  permit  valid  compari- 
sons. It  is  especially  difficult  to  obtain  test  data 
in  time  to  assist  in  decision-making.  Significant 
changes  are  essential  if  OT&E  is  to  realize  its  poten- 
tial for  contributing  to  important  decisions,  particularly 
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where  the  tests  and  the  decisions  must  cross  Service 
lines.  (2) 

The  pressure  continued.  Witnesses  appearing  before  1971  Senate 
and  House  Committees  testified  that: 

...funds  for  testing  and  evaluation  were 
inadequate,  and  organizational  responsibility  was 
fragmented.  At  all  levels  of  management  within 
DOD,  testers  were  not  independent  of  the  devel- 
opers. (3) 

Mr.  G.  W.  Fitzhugh,  Chairman  of  the  Blue  Ribbon  Defense  Panel 
(B.RDP) , addressing  the  operational  testing  problem  at  1971 
hearings  before  the  Senate  Committee  on  Armed  Services,  stated: 

One  of  the  most  urgent  needs  for  improvement  of 
the  entire  weapon  system  acquisition  process  is  more 
effective  operational  test  and  evaluation. 

Everyone  seems  to  agree  that  operational  test 
and  evaluation  (OT&E)  is  very  important;  however, 
there  are  significant  differences  of  opinion  as  to 
what  it  encompasses,  what  its  proper  objectives  are., 
and  what  organizations  and  methods  are  necessary  to 
accomplish  it  most  effectively.  (4) 


Informal  Direction 

The  driving  force  within  DOD  for  changing  T&E  policies  was 
Secretary  Packard,  Deputy  Secretary  of  Defense  under  Melvin 
Laird.  As  a result  of  external  studies  and  his  own  investiga- 
tions, Secretary  Packard  issued  a series  of  memoranda  which 
directed  expedited  change  to  T&E  procedures.  A memorandum, 
dated  11  February  1971,  directed  the  Services  to  arrange  for 

OT&E  to  be  managed  and  conducted  by  an  agency  separate  from  the 
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developing  command  and  which  reports  the  results  of  all  opera- 
tional test  and  evaluation  directly  to  the  Chief  of  the  Service. 
The  memo  also  directed  the  establishment  of  an  OT&E  focal  point 
within  the  Service  headquarters.  In  the  same  document,  Mr. 
Packard  established  a Deputy  Director,  Defense  Research  and 
Engineering,  to  be  the  OSD  focal  point  for  T&E  matters.  (5) 


Formal  Direction 

On  13  July  1971,  the  newly  published  DOD  Directive  5000.1 
officially  tied  OT&E  to  the  DSARC  III  decision  point. 

Test  and  evaluation  shall  commence  as  early  as 
possible.  A determination  of  operational  suitability, 
including  logistic  support  requirements,  will  be  made 
prior  to  large  scale  production  commitments,  making 
use  of  the  most  realistic  test  environment  possible 
and  the  best  representation  of  the  future  operational 
system  available.  The  results  of  this  operational 
testing  will  be  evaluaced  and  presented  to  the  DSARC 
at  the  time  of  the  production  decision.  (6) 

Department  of  Defense  Directive  5000.3,  Test  and  Evaluation, 

dated  January  19,  1973,  provided  additional  and  more  specific 

T&E  policy  guidance  than  DODD  5000.1. 

In  each  DOD  Component  there  will  be  one  major 
field  agency  separate  and  distinct  from  the  develop- 
ing/procuring command  and  from  the  using  command  which 
will  be  responsible  for  OT&E  and  which  will: 

a.  Report  the  results  of  its  independent 
test  and  evaluation  directly  to  the 
Military  Service  Chief  or  Defense  Agency 
Director. 

b.  Recommend  directly  to  its  Military  Service 


8 


>•  'ifBF  K'fS't'SSfW’S.  it 


WgCgrevWTHM— IlIHUi  — 


c. 


Chief  or  Defense  Agency  Director 
the  accomplishment  of  adequate  OT&E. 
Insure  that  the  OT&E  is  effectively 
planned  and  conducted.  (7) 


Official  OSD  direction  for  ITA's  had  been  published 
received  and  understood. 
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SECTION  III 


SERVICES  ESTABLISH  ITA's 

Following  the  release  of  the  Blue  Ribbon  Defense  Panel 
Report  in  mid-1970,  the  Services  took  a more  critical  look  at 
their  past  and  present  RDT&E  activities  with  special  emphasis 
on  OT&E.  Various  studies  were  conducted  based  upon  internal 
criticisms  as  well  as  those  inefficiencies  cited  in  the  BRDP 
Report  and  several  Congressional  Hearing  documents. 

The  Navy 

The  Navy's  problem  of  coming  up  with  a test  agency  independ 
ent  of  the  developer  and  user  was  easier  to  solve  than  those  of 
its  sister  Services.  OPTEVFOR,  which  is  headquartered  in 
Norfolk,  Virginia,  had  been  in  existence  since  1944  when  it  was 
established  for  the  purpose  of  studying  the  Japanese  "Kamikaze" 
threat.  Following  World  War  II,  its  mission  had  been  expanded 
in  the  area  of  tactics  evaluation  and  by  1970  it  was  also  con- 
ducting and  supporting  development  agency  and  user  testing. 
Navy-wide.  To  achieve  the  desired  independence  identified  in 
the  BRDP  Report,  meant  basically  a shift  in  emphasis  to  earlier 
program  involvement  by  OPTEVFOR.  Tnis  change  in  miss. on  orienta 
tion  was  accomplished  in  the  late-1971  to  mid-1972  time  period. 

While  the  reorientation  did  not  involve  an  appreciable  organi- 
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zation  restructuring  or  increased  manpower  allocations , it  did 
require  a different  and  more  expensive  mix  of  personnel.  The 
Navy’s  "hands  on"  operational  testing  concept  demanded  experi- 
enced operational  officers  and  highly  skilled  NCO's  familiar 
with  current  mission , tactics,  policies  and  procedures.  OPTEVFOR's 
current  personnel  strength  totals  1,404  authorizations  which 
includes  310  officers,  868  enlisted,  and  45  Department  of  the  Navy 
civilians.  315  positions  of  the  1,404  authorizations  are  located 
at  OPTEVFOR  Headquarters.  The  remainder  are  spread  across  sub- 
ordinate commands,  aircraft  squadrons  and  detachments  (see  Figure 
3-1).  (8) 

OPTEVFOR's  stated  mission  in  the  area  of  operational  testing 
is  to  plan,  conduct  and  report  on  OT&E  for  all  Navy  acquisition 
programs.  COMOPTEVFOR  reports  OT&E  results  directly  to  the  Chief 
of  Naval  Operations  (CNO)  and  provides  Developing  Agencies  (DA) 
with  all  significant  OT&E  test  data  and  analyses.  OPTEVFOR  is 
also  responsible  for  coordinating/conducting  joint  T&E  and  oper- 
ational testing  for  the  USMC.  Independent  evaluation  reports  on 
USMC  T&E  are  made  directly  to  Commandant  of  the  Marine  Corps  and 
to  the  CNO.  (9) 

OPTEVFOR  is  currently  managing  OT&E  for  219  systems  and  is 
responsible  for  monitoring  OT&E  on  an  additional  209  CNO  assigned 
projects.  Furthermore,  this  ITA  is  providing  DT&E  assistance 
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to  the  DA's  on  approximately  150  development  efforts. 


(10) 


| 

The  Army  j 

I 


Pursuant  to  a letter  from  The  Adjutant  General,  Department 
of  the  Army,  25  September  1972,  the  United  States  Army  Test  and 
Evaluation  Agency  (OTEA)  was  established  effective  that  date  at 
Port  Belvoir,  Virginia.  It  was  activated  under  the  jurisdiction 
of  the  Assistant  Chief  of  Staff  for  Force  Development  (ACSFOR) 
and  a TDA  was  established  with  a personnel  authorization  of  120 
spaces,  which  included  53  officers,  2 enlisted  men,  and  65 
civilians.  (11) 

In  1973,  OTEA  was  reorganized  with  a personnel  authorization 
of  200  spaces  made  up  of  103  officers,  19  enlisted,  and  78  civil- 
ians. Changes  in  1974  made  OTEA  a field  operating  agency  under 
the  Chief  of  Staff,  Army,  and  increased  total  strength  to  223. 

The  most  recent  reorganization  took  place  effective  15  March  1975, 
when  strength  was  again  adjusted  upward  to  OTEA's  present  manning 
of  250  spaces,  which  includes  125  officers,  20  enlisted  men,  and 
105  civilians. 

The  stated  mission  of  OTEA  is  to  support  the  material  acqui- 
sition and  force  development  processes  by  exercising  responsi- 
bility for  all  operational  testing  and  by  managing  force  develop- 
ment testing  (FDTE)  and  joint  user  testing  for  the  Army.  (12) 
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The  OTEA  organization  includes  a command  group,  a coordinating 
staff  and  a functional  staff  (see  Figure  3-2). 

OTEA  is  currently  managing  OT&E  for  22  major  and  17  Category 
I non-major  systems.  In  addition,  the  agency  is  monitoring 
operational  test  activities  on  more  than  250  lesser  category 
non-major  systems.  Out  of  the  pure  OT&E  environment,  OTEA  is 
involved  in  three  major  FDTE  efforts  and  nine  joint  user  testing 
exercises  with  other  Services.  (13) 
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The  Air  Force 

The  Air  Force  was  last  to  formally  establish  their  OSD 
directed  ITA,  doing  so  on  1 January  1974.  Where  the  Army  and 
Navy  had  earlier  complied  explicitly  with  the  BRDP  recommenda- 
tion for  separate,  independent  OT&E  agencies  (OTEA  and  OPTEVFOR) , 
the  Air  Force  initially  attempted  to  achieve  independence  by 
placing  full  OT&E  responsibility  on  the  functional  using  com- 
mands (TAC,  SAC,  etc.).  While  this  complied  with  the  letter  of 
the  BRDP  recommendations  and  in  practice  was  a step  in  the  right 
direction,  it  fell  short  of  the  intent  and  therefore  was  suspect 
and  often  subject  to  criticism.  The  Air  Force  designated  its 
ITA  the  Air  Force  Test  and  Evaluatw  Center  (AFTEC) , located 
this  field  command  agency  at  Kirtland  AFB , New  Mexico,  with 
direct  reporting  to  the  Chief  of  Staff,  Air  Force. 
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Since  becoming  completely  operational  in  October  1974, 

AFTEC' s manning  has  increased  from  an  initial  authorization  of 
208  to  a present  strength  of  231.  This  includes  155  officers, 

32  enlisted  personnel  and  44  civilians.  AFTEC  manning  is  pro- 
jected to  increase  to  290  positions  within  the  next  year. 

The  stated  mission  of  AFTEC  is  to  manage  the  Air  Force 
operational  test  and  evaluation  program  and  assess  the  military 
utility  and  operational  effectiveness  and  suitability  of  major 
and  designated  non- major  systems.  The  center  will  also  actively 
monitor  all  non-major  systems  OT&E  conducted  by  the  major  com- 
mands through  review  and  approval  of  test  plans  and  review  of 
and  comment  on  test  reports.  In  its  role  as  the  manager  for 
Air  Force  OT&E,  AFTEC  will  provide  policy  recommendations  for 
Hq.  USAF  approval  and  subsequent  implementation  by  all  major  air 
commands.  AFTEC  will  act  as  the  Air  Force  spokesman  to  the 
DSARC  on  matters  pertaining  to  OT&E  and  will  support  Congres- 
sional requests  for  OT&E  information  on  weapon  systems  for  which 
procurement  funds  are  being  requested.  (14)  The  AFTEC  organi- 
zation, similar  to  the  Army's  OTEA,  includes  a command  section 
and  coordinating  and  functional  staff  elements  (see  Figure  3-3). 

AFTEC  is  currently  managing  OT&E  for  47  major  and  designated 
non-major  programs.  In  addition,  at  the  present  time  the  center 
is  monitoring  144  major  command  operational  test  efforts  and  is 
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involved  in  nine  OSD-directed  joint  Service  tests.  (15) 


OSD  Officials  Comment 


Hence,  the  Services  achieved  independence  in  systems 
acquisition  testing  by  late  1974.  In  a statement  to  Congress 
in  March  1975,  Lt.  General  Starbird,  USA  (Ret),  then  Deputy 
Director  (T&E) , DDR&E,  remarked: 

Each  of  the  three  Services  now  has  a strong, 
mature  and  effective  organization  for  the  accomplish- 
ment of  independent  operational  test  and  evaluation. 

(They  are  the  Army's  Operational  Test  and  Evaluation 
Agency,  the  Navy's  Operational  Test  and  Evaluation 
Force,  and  the  Air  Force’s  Test  and  Evaluation  Center.) 
(16) 

More  recently,  3 February  1976,  Dr.  Malcom  Currie,  Direct 
of  Defense  Research  and  Engineering,  told  Congress: 

The  past  few  years  have  seen  significant  im- 
provements in  DOD  procedures  for  conducting  T&E  of 
major  weapon  systems.  High  level  emphasis  on  T&E  was 
assured  initially  by  creation  of  the  Office  of  Deputy 
Director  of  Defense  Research  and  Engineering  (Test 
and  Evaluation) . First  under  the  direction  of  Lieuten- 
ant General  Alfred  D.  Starbird,  USA  (Ret.)  and  now  under 
Lieutenant  General  Walter  E.  Lotz,  Jr.,  USA  (Ret.),  this 
office  has  closely  monitored  all  major,  and  many  less- 
than-major,  weapons  acquisition  programs. 

Further  strengthening  of  DOD  T&E  has  been 
achieved  by  the  establishment  of  independent  opera- 
tional T&E  agencies  within  each  of  the  Services.  These 
are  the  Navy's  Operational  Test  a.i  Evaluation  Force, 
the  Army's  Operational  and  Test  Agency  and,  most  re- 
cently, the  Air  Force's  Test  and  Evaluation  Center. 

These  agencies  plan  and  conduct  tests  and  submit  their 
independent  evaluations  directly  to  the  Chiefs  of  thier 
respective  Services.  Their  evaluations  provide  the 
objective  assessment  of  the  operational  effectiveness 
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of  new  weapon  systems  which  is  essential  to  proper 
functioning  of  the  weapons  acquisition  process. 
Creation  of  these  agencies  has  had  a significant 
impact  on  systems  development  and  procurement  as 
evidenced  by  the  greater  attention  being  directed 
to  operational  effectiveness  within  the  Services 
and  by  the  greatly  improved  quality  of  T&E  data 
now  being  presented  to  the  Defense  Systems  Acquisi- 
tion Review  Council  (DSARC) . 

A significant  fallout  from  the  emphasis  on 
T&E  is  a better  definition  of  weapon  system  perform- 
ance goals.  The  T&E  agencies  of  the  Services  and 
the  DD  (T&E)  are  insuring  that  DCP’s  specify  all 
essential  performance  characteristics  and  minimum 
acceptable  values  for  these  parameters  in  advance 
of  undertaking  tests.  This  requires  an  explicit 
definition  of  the  design  parameters  in  operational 
terms  early  in  the  acquisition  cycle.  Such  defini- 
tion not  only  provides  a firm  basis  for  planning 
the  type  and  extent  of  testing  but  also  provides  a 
clear  basis  for  the  evaluation  of  the  military 
utility  of  a new  system.  (17) 
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SECTION  IV 


ITA'S  - SOME  OF  THE  COSTS 
Identifying  Dollar  Costs 

Accurate  identification  of  the  dollar  costs  associated  with 
the  implementation  and  operation  of  the  current  system  of  in- 
dependent testing  (ITA's)  is  extremely  difficult.  It  would  be 
nice  to  be  able  to  report  that  under  the  old  policy,  a system 
OT&E  cost  "X"  dollars,  while  an  OT&E  of  a comparable  system  to- 
day costs  "Y"  dollars.  One  reason  for  the  difficulty  is  that  all 
Services  are  still  involved  in  somewhat  of  a transition  period 
between  the  old  and  new  procedures.  However,  the  primary  reason 
lies  in  the  fact  that  T&E  costs  under  the  previous  policy  of 
joint  developer/user  OT&E  are  not  that  well  documented.  In  other 
words,  no  firm  baseline  for  comparison  exists.  The  various  OT&E 
costs  were  borne  by  different  commands,  different  agencies  within 
the  commands,  and  utilizing  different  sources  and  types  of  funds. 
OT&E  dollars  are  especially  difficult  to  extract  from  development 
contracts  with  industry  where  costs  are  "hidden"  in  labor  (e.g., 
direct  and  indirect),  materials  (e.g.,  prime  system  components, 
spares  and  test  equipment)  and  overhead.  Test  range  costs  in- 
volve both  military  and  commercial  contractor  support  expenses. 
Significant  T&E  dollars  are  expended  by  supporting  and  partici- 
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pating  organizations  such  as  the  using,  maintaining,  logistics 
and  training  commands.  These  costs  are  incurred  over  relatively 
long  periods  of  time,  beginning  during  the  test  planning  phase 
and  extending  through  actual  systems  testing  and  test  report 
submission. 

Under  the  new  policy  where  a single  independent  agency  is 
responsible  for  total  management  of  OT&E,  cost  accounting  should 
be  a more  manageable  function.  However,  there  is  little  indica- 
tion at  this  time  of  any  appreciable  change.  The  true  cost  of 
OT&E  remains  to  be  somewhat  of  an  unknown.  Tnere  are,  however, 
some  facts  and  figures  indicating  that  the  ITA’s  do  represent  an 
increase  in  the  cost  of  RDT&E. 

The  manpower  of  these  new  agencies  is  one  very  evident  cost 
factor.  People  costs,  since  they  currently  account  for  over  50% 
of  the  national  defense  budget,  must  be  considered  when  addressing 
the  price  of  independence.  There  have  been  few  manning  trade- 
offs as  a result  of  ITA  management  of  OT&E.  The  current  person- 
nel strength  of  test  functions  within  program  of rices,  test 
ranges  and  using/supporting  command  elements  do  not  appear  to 
have  been  affected.  In  fact,  from  a systems  program  office  view- 
point, additional  manning  might  be  justified  on  the  basis  that 
the  ITA  represents  another  agency  for  coordination  and/or  ap- 
proval on  all  *-_st  related  activities. 


18 


When  OTEA  and  AFTEC  were  being  established,  the  majority  of 
the  new  personnel  authorizations  were  to  be  achieved  by  a like 

i 

reduction  of  authorized  positions  in  the  other  commands.  How- 
ever, most  of  the  manpower  authorization  trades  resulted  in 

units  giving  up  the  more  expendable  positions.  The  result  of  i 

i 

this  was  that  many  position  conversions  were  required.  For  ex-  s 

t 

ample.-  administrative  and  clerical  positions  had  to  be  converted  j 

f 

to  laboratory  and  other  scientific  specialties.  Likewise,  many  ! 

s 

generalized  enlisted  positions  were  "given  up"  that  required  \ 

conversion  to  specialized  officer  authorizations.  As  has  been  ! 

< 

previously  mentioned,  the  Navy's  OPTEVFOR  OT&E  structure  re-  j 

quired  more  senior  and  more  experienced  officer  capabilities.  ] 

t 

J 

Also,  as  previously  indicated,  there  have  been  manpower  increases  j 

for  both  OTEA  and  AFTEC  since  they  were  established.  Three  \ 

capably  staffed  OT&E  organizations  now  exist  but  at  a significant  \ 

! 

I 

cost,  personnel-wise.  < 

In  the  case  of  the  Army  and  Air  Force,  facilities  for  the 
new  organizations  had  to  be  located.  Once  acquired,  t’'e  attend- 
ant costs  of  outfitting  and  maintaining  these  facilities  is  a 
consideration . 

The  annual  funds  budgeted  for  and  managed  by  the  ITA's 
are  as  indicated  in  the  following  table; 
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TABLE  4-1 


ITA  FUNDING  ($  M) 

OPTEVFOR  OTEA  AFTEC 


FY  76 

FY  77 

FY  76 

FY  77 

FY  76 

FY  77 

O&M 

.57* 

.64* 

5.20 

7.39 

5.67 

5.69 

RDT&E 

3.60** 

4.00** 

3.90 

5.44 

1.64** 

5.64 

*Hqs. 

only  (Does  not 

include  A/C 

Squadrons  & 

Detachments) 

**Does 

not  include  project 

RDT&E  funds. 

Note: 

Table  4-1  does 

not  re/ it  jt 

procurement 

or  RDT&E 

costs  for  test  equipm  "...  a?d  unique  instrumentation 
requirements  (these  cost.:  are  substantial  - for 
example,  OTEA's  estimate  is  $7-10  Million  annually). 


Overtesting? 

A trend  which  is  supported  by  many  officials  involved  at  all 
levels  in  the  systems  acquisition  process  is  that  of  "more  test- 
ing". This  is  particularly  true  in  the  areci  of  OT&E.  However, 
it  is  difficult  to  substantiate  that  this  increased  testing  is 
due  solely  to  the  existence  of  ITA's.  The  extreme  emphasis  on 
testing  prior  to  major  OSD  decisions  has  been  applied  since  1970 
at  every  level  across  DOD.  Additional  systems  testing  is  expen- 
sive and  if  it  has,  in  fact,  reached  the  point  of  "overtesting", 
dollars  are  being  wasted.  There  are  many  military  development 
command  and  certain  OSD  officials  who  believe  and  have  stated 
that  too  much  OT&E  is  now  being  conducted.  One  high  OSD  official 
recent Lv  remarked  that,  "it  appears  that  the  pendulum  of  opera- 
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tional  resting  has  indeed  swung  too  far".  (18)  There  were 
many  reports  of  systems  overtesting.  He  continued  that,  "this 
overtesting  results  in  an  obvious  waste  of  dollars  and  people 
when  what  is  really  needed  is  better  conceived  test  plans".  The 
latter  would  result  in  more  effective  system  evaluation  and  at 
the  same  time  save  critical  dollar  and  people  resources.  (18) 
Overtesting  not  only  impacts  a systems  program  in  dollar 
costs.  It  is  quite  obvious  that  more  testing  normally  involves 
the  procurement  of  additional  test  items  and  other  materials, 
likewise,  increased  testing  clearly  involves  more  time  which 
equates  to  additional  test  range,  using/supporting  command  and 
contractor  personnel  costs.  However,  the  well  used  expression 
"time  is  money"  does  not  go  quite  far  enough  for  the  systems 
program  manager/director.  Since  he  is  the  key  individual  in 
systems  development,  and  totally  responsible  for  a program  from 
concept  through  operational  deployment,  "time  is  more  than 
money."'  The  program  manager  has  a commitment  to  the  user  to 
provide  a fully  tested  weapon  system  by  specific  need  times 
referred  to  as  Initial  Operational  Capability  (IOC)  and  Full 
Operational  Capability  (FOC)  dates.  The  user  in  turn  has  a 
commitment  to  the  Service  and  the  Service  to  the  U.S.  citizen 
in  terms  of  providing  adequate  national  defense.  The  remainder 


of  this  section  will  discuss  some  of  these  non-dollar  co^ts  of 


ITA‘ s as  seen  through  the  eyes  of  a DOD  program  manager  (PM) . 

Non-Dollar  Costs 

The  introduction  of  ITA's  into  FDT&E  has  taken  from  the 
PM  a significant  portion  of  his  control  of  the  program.  He  still 
has  total  program  responsibility,  but  no  longer  has  the  authority 
with  which  to  control  a major  function  of  development,  that  of 
OT&E.  He  has  lost  the  flexibility  he  once  possessed  through 
OT&E  planning  and  budgeting.  Trade-offs  between  testing  and 
other  phases  of  system  acquisition  can  no  longer  be  made  within 
the  program  management  office  (PMO) . 

The  ITA  also  represents  another  agency  with  which  the  PMO 
must  interface.  In  the  current  transition  period,  it  has  been 
especially  difficult  and  time  consuming  to  bring  the  ITA's  "up 
to  speed"  in  order  that  they  possess  the  necessary  knowledge  and 
capability  to  assume  full  OT&E  management.  For  future  programs, 
this  problem  should  be  minimal  since  the  ITA  would  have  been  "on 
board"  since  the  conceptual  phase  of  the  programs.  However,  the 
interfacing  requirements  will  continue  to  be  significant  since 
the  ITA  represents  not  only  another  coordination  agency,  but  also 
an  action  agency  on  all  test  plans  and  major  decision  documents 
that  impact  the  DSARC  reviews. 

The  existence  of  the  ITA  in  its  position  of  OT&E  test  manag- 
er has  also  resulted  in  the  constriction  of  information  flow 


between  the  PMO  and  the  user;  said  another  way,  between  the 
developer  and  the  customer.  The  PM  regards  this  loss  of  impor- 
tant dialogue  as  another  cost  of  control  since  the  critical 
feedback  loop  has  been  severed  or  at  least  interrupted.  A like 
situation  has  developed  between  the  PMO  and  the  actual  test 
agency,  be  it  a government  or  Service  test  range/facility. 

The  FM's  of  lesser  priority  programs  feel  that  they  stand 
to  lose  the  most  control  due  to  the  large,  and  what  appears  on 
the  surface  to  be  an  unwieldy,  number  of  programs  being  managed 
r»r  monitored  by  the  ITA's.  They  anticipate  that  the  higher 
priority  programs  will  receive  more  attention  and  by  more  senior 
and  more  experienced  test  managers. 

Probably  the  most  significant  cost  of  control,  or  loss  of 
PM  control,  exists  because  of  the  location  of  the  ITA's  in  the 
military  hierarchy.  These  agencies  report  directly  to  the 
Service  Chiefs.  In  addition,  and  as  one  ITA  mission  statement 
reflects,  "AFTEC  is  the  official  Air  Force  spokesman  to  the 
DSARC...and  supports  Congressional  requests .. .on  weapon  systems 
for  which  procurement  funds  are  being  requested".  (15)  This 
represents  an  unprecedented  loss  of  program  control  from  the  PM 
to  what  is  in  fact  a higher  level  agency.  In  -essence,  the  ITA 
activity  can  be  regarded  as  a significant  step  backward  in  OSD's 
long-standing  policy  of  decentralizing  systems  acquisition 


management. 

The  costs  included  in  this  section  by  no  means  form  a 
complete  list.  They  do,  however,  represent  areas  that  should 
be  addressed  when  and  if  a formal  reassessment  of  ITA's  is 
accomplished. 
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SECTION  7 


SUMMARY 


During  the  1970-71  time  period,  a President's  panel  and 
Congress  expressed  the  need  for  additional  test  and  evaluation 
prior  to  DOD  major  decisions  on  weapon  systems  acquisitions. 

The  need  was  further  defined  as  being  twofold  in  nature.  First, 
there  was  a requirement  for  an  increased  quantity  of  system 
testing  and  associated  test  data.  And  secondly,  there  was  a 
need  for  more  objective  testing  by  separating  the  tester  from 
the  developer  and  user.  In  response,  OSD  directed  'DODD  5000.1 
and  5000.3)  the  Services  to  establish  independent  test  agencies 
to  manage  all  weapon  systems  OT&E.  In  addition,  it  was  directed 
that  these  agencies  report  OT&E  test  results  directly  to  the 
Service  Chiefs. 

By  1974,  all  three  services  had  ITA's  operating  and  either 
managing  or  monitoring  all  OT&E  programs.  The  Army  and  the  Air 
Force  agencies  were  and  continue  to  be  quite  similar  in  size  and 
mission.  The  Navy  had  a test  agency  in  existence  which  was  near- 
ly independent  of  the  developer  and  user;  through  mission  re- 
orientation and  some  restructuring,  it  assumed  the  full  OT&E 
management  responsibility. 

The  costs  associated  with  the  implementation  and  operation 


of  the  ITA's  are  substantial.  However,  due  to  the  lack  of  a 
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baseline  for  comparison,  it  is  difficult  to  assess  how  much  more 
costly  the  new  ITA  program  is  from  the  previous  policy  of  joint 
developer/user  testing.  OT&E  cost  accounting  was  and  remains 
very  difficult  because  of  the  number  of  organizations  involved 
in  OT&E  and  the  various  types  and  sources  of  funding  utilized. 

There  is  evidence  indicating  that  because  of  the  emphasis 
placed  on  more  testing  by  the  BRDP,  Congress  and  OSD,  the 
Services  have  overreacted.  The  pendulum  may  have  swung  too  far, 
to  the  point  where  additional  testing  has  become  non-cost  effec- 
tive. 

As  seen  by  a DOD  Program  Manager,  costs  resulting  from  the 
introduction  of  ITA's  are  more  than  just  dollar  oriented.  As 
significant  are  the  losses  in  control  of  his  program,  a usurping 
of  his  implied  authority  as  the  individual  charged  with  total 
program  management  responsibility. 
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SECTION  VI 
CONCLUSIONS 

Through  the  establishment  of  ITA's,  the  DOD  systems  acquisi- 
tion process  has  incurred  an  abrupt  change  to  the  manner  in  which 
systems  OT&E  is  managed,  conducted  and  reported.  With  any  abrupt 
change  in  policy,  there  is  often  a tendency  on  the  part  of  the 
directing  agency  to  overemphasize  the  change  and  the  reasons  for 
it.  This  overemphasis  in  turn  leads  to  an  overreaction  by  the 
organizations  upon  which  the  policy  was  directed. 

It  is  quite  possible  that  such  a cause  and  effect  relation- 
ship existed  with  the  direction  for  more  OT&E  and  the  implementa- 
tion of  ITA's.  Evidence  supports  the  fact  that  the  new  OT&E 
policy  has  resulted  in  higher  RDT&E  costs.  However,  complete 
substantiation  of  this  is  only  possible  by  first  improving  T&E 
cost  accounting  systems  within  all  Services. 

Last,  but  not  least,  the  ITA  concept  has  been  costly  to  DOD 
program  managers  in  terms  of  proper  program  management  control. 


SECTION  VII 


RECOMMENDATIONS 

Three  significant  T&E  issues  have  emerged  as  a result  of 
this  research.  The  issues  are  interdependent,  as  are  the  recom- 
mendations for  their  resolution. 

The  first,  issue  is  the  inadequacy  of  the  military  services' 
T&E  cost  accounting  procedures.  This  factor  also  makes  the  T&E 
budgeting  processes  themselves  suspect.  It  i.s  recommended  that 
the  services  take  a "hard  look"  at  their  T&E  budgeting  and  cost 
accounting  procedures,  identify  the  weaknesses,  and  institute 
improved  financial  control. 

The  second  issue  is  that  of  possible  overtesting  during 
system  OT&E.  Tne  Services  should  analyze  their  current  OT&E 
planning  at  all  levels  to  insure  that  only  the  minimum  necessary 
is  being  planned  and  accomplished. 

The  third  and  major  issue  is  the  need  for  ITA's.  Recommend 
that  OSD  and  the  Services  reassess  and  reverify  the  rsquirement. 
f )r  independent  testing  as  it  is  currently  conducted.  This 
assessment  could  be  based  upon  those  system-oriented  factors  that 
the  DOD  Program  Manager  considers  daily;  namely,  cost,  schedule, 
and  performance. 
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